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ABOUT  ABSVal 


•  WHICH  SIMULATIONS  are  the  target  of  interest 

■  Directly  applicable  to  IW  problem  set 

■  In  addition  to  ISAAC,  Pythagoras,  MANA,  consider  decision 
rules,  knowledge-based  systems,  cellular  automata, 
population  dynamics 

•  Discussion  about  W&A 

■  Goal  of  validation:  Match  Tool  to  Application 

•  Conservation  of  Vagueness:  definition  of  “match” 

•  DoD  definitions  &  Processes 

•  Looking  at  UK  MOD,  AIAA,  ASME,  NASA,  DOE 

•  Two  phases 

■  This  phase:  What  we  need  to  know  to  evaluate  an  ABS 

■  Next  Phase:  Experiment  with  methods 
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•  Conservation  of  Vagueness:  definition  of  “mafcft” 

DoD  definitions  &  Processes 

•  Looking  at  UK  MOD,  AIAA,  ASME,  NASA,  DOE 

Two  phases 


■  This  phase:  What  we  need  to  know  to  evaluate  an  ABS 

■  Next  Phase:  Experiment  with  methods 


Schism 


Agent-based  simulations  use  modular  rules 
and  local  reasoning  to  produce  realistic 
and/or  interesting  emergent  aggregate 
behavior. 

■  Surprise  is  good** 


Successful  simulation  testing  (core  to 
face/results  validation)  based  on 
demonstrating  credibility  across  the  range  of 
potential  input. 


■  Surprise  not  good** 


**  Refined  later  in  this  talk 


Questions 

*  What  activities  can  I  reasonably  support  with  my 
ABS? 

*  What  are  the  limits? 

*  What  caveats  are  necessary? 

*  Compared  with  traditional  simulation  solutions,  how 
are  my  results  to  be  used? 

*  How  can  I  make  credibility  statements  about  a 
simulation  that  is  out  of  my  (top-down)  control? 

■  Value  of  training  experiences 

■  Value  of  analytical  results 


•  Can  I  support  the  scientific  method  with  this  ABS? 


ABSVal  Products 


•  General,  institutionally  acceptable 
processes  and  criteria  for  assessing 
the  validity  of  an  agent-based 
simulation  used  as  part  of  a  DoD-level 
analysis 

■  What  information? 

■  What  assurances  and  endorsements? 

■  What  desirable  qualities? 


Benefit 


•  Increased  awareness  of  the  value  of 
analysis  results  supported  by  agent  based 
simulation(s) 

•  [Potential]  Increased  credibility  of  results 

•  [Potential]  More  valuable  agent-based 
simulations 

•  [Potential]  Responsible  analytical 
application  of  ABS  by  OA  professionals 

•  [Potential]  Civilization  of  the  discourse 
concerning  ABS-generated  analysis  results 


Benefactors:  HBR,  VVA,  All  Military  Organizations  using  ABS,  Analysis, 
Planning,  Experimentation,  Training,  Acquisition,  ... ' 


First  Principles 


GOAL  OF  SIMULATION 

a)  Aggregate  effects  you 
understand 

b)  Calculate  probability  of 
simultaneous/sequential 
events 

c)  Challenge  user’s  intuition** 


SEEING  THE  INSIDE  AND  OUTSIDE 


DATA 

a)  Model  exists  because  of  the  data 

b)  Data  exists  because  of  the  model 

c)  Accuracy,  precision 

d)  Covering  the  possible  truths 

e)  Propagating  uncertainty  &  model 
sensitivity  analysis 


VALIDATION  SURROGATES 


a)  Depict  agent’s  behavior 

b)  Depict  aggregation  methods 

c)  Serial  aggregation  (building 
blocks) 

d)  Prose,  pictures,  diagrams,  tests 

e)  Visualization  of  outcomes, 
trends,  cause-effect 


a)  History  of  successful  uses 

b)  Credible  believers 

c)  Large,  mature  user  community 

d)  Transparency 

e)  Relative  validity 

f)  (Over-?)  Fitting  historical  data 


Surprise! 


Surprise 


Accept/reject 


Explore 


Production 

Runs 


Explain 


What  abstractions,  data  values,  shortcuts,  tricks,  intentional  oversights, 
modifications  to  expectations,  code  changes  were  required? 


Representation 


Dynamic  Influence 

■  Ist-order  effect 

■  Direct  influence 

■  Relevant  over  large 
interval 

■  Plausibly  relevant 
over  limited  interval 

■  Possibly  influential 

■  Minor  detail 

■  No  relevance 


•  Distillation 

■  Include  only  the 
highly-relevant 
dynamics 

■  Aggregation  of 
effects 

■  Referent  often 
loose/missing 


Completeness 
vs.  Parsimony 


Statistical  Methods 

Balance  Predictiveness  vs.  Parsimony 

Xj’s  are  the  levels  of  dynamics  included/ 
excluded  (capacities) 

Y  is  the  response  variable  (utility) 


Qualitative 

assessment 

meets 

Critical  Values 


Bailey,  M.  P.  and  W.  G.  Kemple.  1994.  The  Scientific  Method  of 
Choosing  Model  Fidelity,  WSC  Proceedings. 


Goals 


•  Understanding  the  meaning  of  Valid  Enough 

•  Techniques  for  uncovering  validation 
shortcomings 

■  in  the  presence  of  a  weak  referent 

•  Expressing  validation  boundaries 

•  Being  conservative  with  VV&A  resources 

•  Framework 

■  transparent,  traceable,  repeatable,  communicate-able 


In  Sum 


•  Achieve  the  Goals  of  Simulation 
Validation  for  ABS 

■  Concentrate  on  analytical  applications 

■  Test-case-driven  &  practical 

■  Institutional  acceptability 

■  Vast  collection  of  potential  partners 


